54 research outputs found
Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection
Linguistically diverse datasets are critical for training and evaluating
robust machine learning systems, but data collection is a costly process that
often requires experts. Crowdsourcing the process of paraphrase generation is
an effective means of expanding natural language datasets, but there has been
limited analysis of the trade-offs that arise when designing tasks. In this
paper, we present the first systematic study of the key factors in
crowdsourcing paraphrase collection. We consider variations in instructions,
incentives, data domains, and workflows. We manually analyzed paraphrases for
correctness, grammaticality, and linguistic diversity. Our observations provide
new insight into the trade-offs between accuracy and diversity in crowd
responses that arise as a result of task design, providing guidance for future
paraphrase generation procedures.Comment: Published at ACL 201
Tuning the Diversity of Open-Ended Responses from the Crowd
Crowdsourcing can solve problems that current fully automated systems cannot.
Its effectiveness depends on the reliability, accuracy, and speed of the crowd
workers that drive it. These objectives are frequently at odds with one
another. For instance, how much time should workers be given to discover and
propose new solutions versus deliberate over those currently proposed? How do
we determine if discovering a new answer is appropriate at all? And how do we
manage workers who lack the expertise or attention needed to provide useful
input to a given task? We present a mechanism that uses distinct payoffs for
three possible worker actions---propose,vote, or abstain---to provide workers
with the necessary incentives to guarantee an effective (or even optimal)
balance between searching for new answers, assessing those currently available,
and, when they have insufficient expertise or insight for the task at hand,
abstaining. We provide a novel game theoretic analysis for this mechanism and
test it experimentally on an image---labeling problem and show that it allows a
system to reliably control the balance betweendiscovering new answers and
converging to existing ones
Captions versus transcripts for online video content
ABSTRACT Captions provide deaf and hard of hearing (DHH) users ac cess to the audio component of web videos and television. While hearing consumers can watch and listen simultane ously, the transformation of audio to text requires deaf view ers to watch two simultaneous visual streams: the video and the textual representation of the audio. This can be a prob lem when the video has a lot of text or the content is dense, e.g., in Massively Open Online Courses. We explore the ef fect of providing caption history on users' ability to follow captions and be more engaged. We compare traditional onvideo captions that display a few words at a time to off-video transcripts that can display many more words at once, and investigate the trade off of requiring more effort to switch be tween the transcript and visuals versus being able to review more content history. We find significant difference in users' preferences for viewing video with on-screen captions over off-screen transcripts in terms of readability, but no signifi cant difference in users' preferences in following and under standing the video and narration content. We attribute this to viewers' perceived understanding significantly improving when using transcripts over captions, even if they were less easy to track. We then discuss the implications of these re sults for on-line education, and conclude with an overview of potential methods for combining the benefits of both onscreen captions and transcripts
The Effects of Sequence and Delay on Crowd Work
A common approach in crowdsourcing is to break large tasks into small microtasks so that they can be parallelized across many crowd workers and so that redundant work can be more easily compared for quality control. In practice, this can re-sult in the microtasks being presented out of their natural order and often introduces delays between individual micro-tasks. In this paper, we demonstrate in a study of 338 crowd workers that non-sequential microtasks and the introduction of delays significantly decreases worker performance. We show that interruptions where a large delay occurs between two related tasks can cause up to a 102 % slowdown in com-pletion time, and interruptions where workers are asked to perform different tasks in sequence can slow down comple-tion time by 57%. We conclude with a set of design guide-lines to improve both worker performance and realized pay, and instructions for implementing these changes in existing interfaces for crowd work. Author Keywords Crowdsourcing; human computation; workflows; continuity
Real-time crowd control of existing interfaces
Crowdsourcing has been shown to be an effective approach for solving difficult problems, but current crowdsourcing systems suffer two main limitations: (i) tasks must be repackaged for proper display to crowd workers, which generally requires substantial one-off programming effort and support infrastructure, and (ii) crowd workers generally lack a tight feedback loop with their task. In this paper, we introduce Legion, a system that allows end users to easily capture existing GUIs and outsource them for collaborative, real-time control by the crowd. We present mediation strategies for integrating the input of multiple crowd workers in real-time, evaluate these mediation strategies across several applications, and further validate Legion by exploring the space of novel applications that it enables
StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible
Blind people frequently encounter inaccessible dynamic touchscreens in their
everyday lives that are difficult, frustrating, and often impossible to use
independently. Touchscreens are often the only way to control everything from
coffee machines and payment terminals, to subway ticket machines and in-flight
entertainment systems. Interacting with dynamic touchscreens is difficult
non-visually because the visual user interfaces change, interactions often
occur over multiple different screens, and it is easy to accidentally trigger
interface actions while exploring the screen. To solve these problems, we
introduce StateLens - a three-part reverse engineering solution that makes
existing dynamic touchscreens accessible. First, StateLens reverse engineers
the underlying state diagrams of existing interfaces using point-of-view videos
found online or taken by users using a hybrid crowd-computer vision pipeline.
Second, using the state diagrams, StateLens automatically generates
conversational agents to guide blind users through specifying the tasks that
the interface can perform, allowing the StateLens iOS application to provide
interactive guidance and feedback so that blind users can access the interface.
Finally, a set of 3D-printed accessories enable blind people to explore
capacitive touchscreens without the risk of triggering accidental touches on
the interface. Our technical evaluation shows that StateLens can accurately
reconstruct interfaces from stationary, hand-held, and web videos; and, a user
study of the complete system demonstrates that StateLens successfully enables
blind users to access otherwise inaccessible dynamic touchscreens.Comment: ACM UIST 201
- …